Evidence Of Internal Validity Research Articles

Evaluating methodological enhancements to the Yes/No Angoff standard-setting method in language proficiency assessment

This study evaluated the efficacy of two proposed methods in an operational standard-setting study conducted for a high-stakes language proficiency test of the U.S. government. The goal was to seek low-cost modifications to the existing Yes/No Angoff method to increase the validity and reliability of the recommended cut scores using a convergent mixed-methods study design. The study used the Yes/No ratings as the baseline method in two rounds of ratings, while differentiating the two methods by incorporating item maps and an Ordered Item Booklet, each of which is an integral tool of the Mapmark and the Bookmark methods. The results showed that the internal validity evidence is similar across both methods, especially after Round 2 ratings. When procedural validity evidence was considered, however, a preference emerged for the method where panelists conducted the initial ratings unbeknownst to the empirical item difficulty information, and then such information was provided on an item map as part of the Round 1 feedback. The findings highlight the importance of evaluating both internal and procedural validity evidence when considering standard-setting methods.

Development and validation of a vaccine hesitancy scale for COVID-19 to the Portuguese language

This is a psychometric study aimed at developing and validating a specific scale that assesses the vaccine hesitancy behavior of COVID-19 to the Portuguese language. The vaccine hesitancy scale for COVID-19 vaccines was developed in Brazil based on the Vaccine Hesitancy Scale developed by the Strategic Advisory Group of Experts on Immunization Working Group. Data were collected at the end of the first wave of the COVID-19 pandemic in Brazil (between March 1 and June 30 of 2021) by convenience sampling and snowballing using electronic surveys with 1345 participants. We carried out an exploratory factorial analysis (EFA) and confirmatory factorial analysis (CFA) of 10 items. Our findings for the EFA suggested only a single dimension and the unidimensional congruence of 0.95 and explained a common variance of 0.90; all items presented a significant factor loading (≥ 0.30) with excellent reliability (Cronbach’s alpha of 0.88). The CFA showed the model fit indices were satisfactory (x2 = 66.0538, gl = 35, p = 0.001; RMSEA = 0.03 CI [0.02;0.04]); CFI = 0.98; TLI = 0.98) with the factor loadings cut-off ranging from 0.32 to 0.88. Our findings revealed that the vaccinal hesitancy assessment tool for COVID-19 vaccines in the Portuguese language has evidence of internal validity and reliability, and it is best represented as a unifactorial 10-item structure yielding a total score.

Psychometric properties of the Latino Students Patient Safety Questionnaire, Brazilian version.

to assess the psychometric properties of the Latino Students Patient Safety Questionnaire, Brazilian version. a methodological study, carried out between April 2020 and January 2021, with 218 nursing and medicine students. Structural and discriminant construct validity were assessed by confirmatory factor analysis and cross factor loadings. Reliability was verified by McDonald's omega, average variance extracted, composite reliability, and item-total correlation. the final model reproduced the original structure of 21 items, distributed in five dimensions, requiring the exclusion of one item. Acceptable fit indexes were obtained (x2/gl=2.325; CFI=0.99; TLI=0.98; RMSEA=0.054). Discriminant validity was confirmed. Reliability indicators were adequate, except McDonald's omega, in one factor (0.68), and average variance extracted, in two factors (0.41; 0.47). the instrument demonstrated evidence of internal validity and satisfactory reliability among nursing and medical students.

Internal validity evidence of a knowledge questionnaire on alcohol use, abuse and dependence and associated questions.

to perform content validity and verify the psychometric properties of the adapted version of an alcohol knowledge questionnaire and associated questions. a methodological study, in which a committee of judges analyzed the questionnaire item representativeness, clarity and relevance. Item response theory was used to assess the instrument psychometric properties applied to a sample of 240 health professionals. the questions were adjusted according to judges' assessment, obtaining a satisfactory Content Validity Index (0.98). High discrimination ability and adequate difficulty levels were observed in 75% of multiple-choice questions and in 25% of statements. the instrument presented content validity with satisfactory indices. However, it is recommended that the questionnaire adapted in this study be used in different samples of health professionals from other parts of Brazil, in order to provide greater robustness to its reliability.

Internal validity evidence of a knowledge questionnaire on alcohol use, abuse and dependence and associated questions

RESUMO Objetivos: realizar a validação de conteúdo e verificar as qualidades psicométricas da versão adaptada do questionário de conhecimentos frente ao álcool e questões associadas. Métodos: estudo do tipo metodológico, em que um comitê de juízes analisou a representatividade, clareza e pertinência dos itens do questionário. Utilizou-se a teoria da resposta ao item para avaliar as qualidades psicométricas do instrumento aplicado em uma amostra de 240 profissionais de saúde. Resultados: as questões foram ajustadas conforme avaliação dos juízes, obtendo um Índice de Validade de Conteúdo satisfatório (0,98). Observou-se alta capacidade de discriminação e níveis de dificuldade adequados em 75% das questões de múltipla escolha e em 25% das afirmativas. Conclusões: o instrumento apresentou validade de conteúdo com índices satisfatórios. Entretanto, recomenda-se que o questionário adaptado neste estudo seja utilizado em amostras diversas de profissionais de saúde, de outros locais do país, com o intuito de oferecer maior robustez à sua confiabilidade.

Self-Determination assessment and evaluation in Romanian adolescents: preliminary psychometric properties of the ARC’s self-Determination scale

ABSTRACT Research in self-determination has expanded in recent years, especially in the special education field. Instruments to measure self-determination in students with disabilities are not available in the Romanian language, and this issue has led to the development of the current study, a cultural adaptation , and validation of The ARC’s Self-Determination Scale which provides a strong psychometric measure of self-determination. Based on the Functional Model of Self-determination, this scale is designed to measure the four components of self-determination construct and global self-determination in students with disabilities. This pilot test study aims to explain the cultural adaptation and validation process of The ARC’s Self-Determination Scale for use in the Romanian research context and the educational approach. The participants’ sample for the validation process of the scale comprised 275 young students with learning disabilities, and mild and moderate intellectual disabilities, the participants were ranging ages from 11 and 19 years old (M = 14.93; SD = 1.47). Through the structuraequation modeling approach, the estimation of the latent model fit provided evidence of internal validity of the scale, demonstrating the adequacy to measure self-determination in adolescents with mild and moderate intellectual disabilities. The values of internal consistency coefficients and validity characteristics are discussed. .

Evidence of internal validity of the social skills inventory for caregivers (SSI-CE) in Brazil

With population aging, it is important to develop studies on psychological skills that can facilitate the task of caregivers of a dependent older adult. The Social Skills Inventory for Caregivers of the Elderly (SSI-CE) is an assessment tool for this purpose. Even considering that the instrument already has evidence of validity, the aim of this study is to seek additional evidence of validity for the SSI-CE. To this end, 533 family caregivers of older adults with an average age of 49.7 years were investigated by responding to the following instruments: Sociodemographic Questionnaire, Brazil Economic Classification Criteria, and SSI-CE. Initially, two factor models for the SSI-CE were compared (second-order and bifactor models). Then, analyses were performed for measurement invariance and sample heterogeneity of the SSI-CE in relation to the variables age, gender, presence of dementia in the elderly care receiver, and type of data collection (online or in-person). Finally, the instrument's reliability values were checked. After comparing the models, the second-order model was the most appropriate, as it displayed reliability values ranging from ω = 0.66 to ω = 0.89. In invariance analysis, variability was found for sample heterogeneity in the variables gender, presence of dementia, and type of data collection. In conclusion, the calculation of a global score for the SSI-CE is pertinent, with good evidence of internal validity. However, specific rules should be established, considering different sample characteristics.

Phenomenological assessment of psychedelics induced experiences: Translation and validation of the German Challenging Experience Questionnaire (CEQ) and Ego-Dissolution Inventory (EDI).

Several measures have been designed to assess subjective experiences induced by psychedelic substances or other mind-altering drugs as well as non-pharmacological methods. Recently, two self-report questionnaires have been introduced to measure acute adverse effects following psilocybin ingestion and the phenomenon of ego-dissolution associated with the use of psychedelics, respectively. The 26-item Challenging Experience Questionnaire assesses multiple facets of psilocybin induced experiences on seven subscales, whereas the 8-item Ego-Dissolution Inventory consists of a unidimensional scale. In the present study, these questionnaires were translated into German and their psychometric properties then evaluated in an online survey on psychedelics induced experiences. Confirmatory factor analysis suggested an acceptable fit of the 7-factor structure of the German Challenging Experience Questionnaire with overall good internal consistency for all subscales. The factor structure did not differ based on gender or prior struggle with a psychiatric disorder, furthering the evidence of internal validity. Correlations with the State-Trait-Anxiety Inventory and the Altered States of Consciousness Rating Scale demonstrated convergent validity. Confirmatory factor analysis did not support the hypothesized single-factor structure of the German Ego-Dissolution Inventory and exploratory factor analysis suggested an alternative factor structure, where only five items loaded onto a common factor that was interpreted as ego-dissolution. Internal consistency of this 5-item measure was high and correlation with selected items of the Mystical Experience Questionnaire and Altered States of Consciousness Rating Scale supported convergent validity. Translation and validation of these questionnaires contribute to the advancement of common standards in the psychological and neuroscientific study of altered states of consciousness.

The Job Demands Scale for Hearing-Impaired Employees: Development and preliminary analysis of internal validity

Few work participation scales have been normed on employees with hearing impairment. This creates an employment disparity against them. We developed a 57 item Job Demands for Hearing Impaired Employees (JDHIE) scale, providing preliminarily internal validity evidence. Respondents were 85 South African adults with hearing impairment from a variety of occupational settings (female = 45.9%; deaf = 42.4%). Exploratory Factor Analysis (EFA) findings yielded a three-factor scale for communication, environmental, and time demands. The reliability of scores from the communication, environmental and time demands were high, providing the preliminarily evidence for further validation studies of this instrument.

Validation of the 12-item HIV-stigma scale adapted for use among Tunisian COVID-19 patients

BackgroundAlthough, there is a rapidly accumulating body of studies on the various aspects of COVID-19, little interest was given to develop valid specific scales measuring COVID-19-related stigma. The current study aims to testvalidity and reliability ofa modified version of the 12-item HIV Stigma Scale, adapted for adults with COVID-19 infection living in the region of Sousse (Tunisia).MethodsA cross sectional study was conducted among patients with COVID-19 infection between November and December 2020. The 12-item HIV stigma scale was translated in Arabic and then adapted to the Tunisian context. A preliminary version with twenty items covering the four dimensions of COVID-19 stigma was tested (Personalised stigma, disclosure concerns, concerns with public attitudes and negative self-image). The psychometric evaluation included internal consistency as well as principal component analysis (PCA).ResultsResponses of 346 COVID-19 patients were obtained.Among participants, 60.4% were women with a median age of 40 years (interquartile range, 30 to 54.5 years). Evidence for internal validity was supported by a PCA, suggesting a three factor solution with fourteen items. The 14-item scale demonstrated good internal consistency. The global Cronbach'sα was 0.91, with values of 0.94, 0.93 and 0.98 for social stigma, negative self-image and disclosure concerns respectively.ConclusionsThe results suggest construct validity and reliability of the 14-item stigma scale measuring three aspects related to COVID-19 infection in Tunisia.Key messages The 14-item stigma scale is a good psychometric tool to be used in assessing COVID-19 related stigma in Tunisia.Monitoring COVID-19 stigma with a valid and reliable would guide its prevention actions.

Development and Psychometric Validation of the Brief Nurses' Practice Environment Scale and Its Relation to Burnout Syndrome and Job Satisfaction: A Study in Spanish Nurses.

Introduction: Nursing environment is a vast concept that traditionally has included a wide range of job characteristics and has been related to burnout and job satisfaction. For its measurement, the Practice Environment Scale of the Nursing Work Index (PESNWI) stands out. However, shorter instruments are needed. The purpose of the study is to develop and test the Brief Nurses' Practice Environment (BNPE) Scale.Methods: The BNPE Scale was developed and tested in a sample of 210 Spanish nurses (data collection 2018).Results: Cronbach's alpha was 0.702. The confirmatory factor analysis (CFA), with an excellent fit, offered evidence of internal validity. Regarding validity, the BNPE Scale predicted both burnout and job satisfaction. Finally, evidence pointed out a cutoff score of <12 for low levels of practice environment and a cutoff score of >15 for higher levels in practice environment.Conclusions: The BNPE Scale is a short, easy-to-use measure that could be employed in major batteries assessing the quality of healthcare institutions.

Setting and Validating Multiple Standards on a Multistage‐Adaptive Test

AbstractSetting cut scores on multistage‐adaptive tests (MSTs) is difficult, particularly when the test spans several grade levels, and the selection of items from MST panels must reflect the operational test specifications. In this study, we describe, illustrate, and evaluate three methods for mapping panelists’ Angoff ratings into cut scores on the scale underlying an MST. The results suggest the test characteristic function and item characteristic curve methods performed similarly, but the method based on dichotomizing panelists’ ratings at a response probability of .67 was unacceptable. The study featured a rating booklet design that allowed us to systematically evaluate the validity of the Angoff ratings across test levels, which contributed internal validity evidence for the cut scores, which were also evaluated using procedural and external validity evidence. The implications of the results for future standard setting studies and research in this area are discussed.

Quality versus Risk-of-Bias assessment in clinical research

Assessment of internal validity safeguards implemented by researchers has been used to examine the potential reliability of evidence generated within a study. These safeguards protect against systematic error, and such an assessment has traditionally been called a quality assessment. When the results of a quality assessment are translated through some empirical construct to the potential risk of bias, this has been termed a risk of bias assessment. The latter has gained popularity and is commonly used interchangeably with the term quality assessment. This key concept paper clarifies the differences between these assessments and how they may be used and interpreted when assessing clinical evidence for internal validity.

The development and validation of an empirical scale to inform community diagnostics and change

ABSTRACT Purpose The purpose of this research was to develop and validate a robust scale, based on the community capital framework, to provide an empirical measure of community perceptions. Design/methodology/approach The proposed measure was completed by a sample of Georgia citizens (n = 3374) representing a diverse cross-section of community characteristics. Findings Data indicate that the six proposed scales (natural, social, human, cultural, built-financial, and political) subsumed within the instrument measure unique underlying latent variables with evidence of internal validity through multiple analytic techniques. Practical implications This study represents an initial examination of a quantitative instrument – the Community Diagnostics + Social Impact (CD + SI) ToolkitTM – that can be used by researchers, educators, and community development professionals to measure the stocks of the various capitals within a community. Theoretical implications The theoretical implications of this study are that the CD + SI ToolkitTM holds promise as a valid and reliable instrument for establishing holistic community profiles, identifying targeted areas for in-depth research inquiry or programmatic design, guiding policy design and implementation, and ultimately evaluating the impacts of community development programs and policies. The CD + SI ToolkitTM addresses an omission in the literature and in broader community development. Originality/value While various frameworks such as the CCF exist to provide a more general blueprint for program development and assessment, the CD + SI ToolkitTM represents an effort to specifically quantify the community capitals and establish a replicative process for measuring community capitals. The results of this initial validation effort demonstrate the potential for such a tool.

Comportamiento innovador en el trabajo: Análisis factorial confirmatorio de la Escala de Janssen

Innovative work behaviour is defined as the intentional creation, introduction and application of new ideas within a work role, group or organization, in order to benefit role performance, the group, or the organization. Following Janssen (2000), it is conceive as complex behaviour consisting of a set of three different behavioural tasks: idea generation (the production of novel and useful ideas in any domain), idea promotion (the socialization of ideas and built of a coalition of supporters among potential allies such as friends and sponsors, who provide the necessary power and resources to develop the idea), and idea realization (the production of a prototype or model of the innovation that can be experienced and ultimately applied). Although innovative work behaviour is theoretically treated as multi-dimensional, available measures are mostly one-dimensional. Also, the empirical evidence for the validity of innovative work behaviour measures is limited. The Innovative Work Behavior Scale of Janssen (2000) stands out as one of the most measures used in contemporary research. However, there is still no consensus regarding its uni or tri-factorial structure. This study presents evidence of internal validity of the Janssen´s Innovative Work Behavior Scale. An instrumental study was designed on a multi-oupational sample of 543 workers (64 % women, M age = 35.32 years, labor M seniority = 8.78 years). Innovative work behaviours were measured with an argentinean adaptation of Janssen ´s scale. The scale consist in 9 item and it was completed by the own respondents (self-report). The response format was a 5-point scale ranging from 'never' (1) to 'always' (5). Confirmatory factorial analysis were conducted to examinate the scale´s dimensionality. Three measurement models (based on theory and empirical evidence) was estimated: (a) a uni-factorial model, (b) an oblique tri-factorial model and, (c) a tri-factorial model with a second-order factor. Values of absolute, incremental and parsimonious fit were within adequated ranges. Nevertheless, the correlation coefficients between the latent factors indicated problems in the estimation of the last two models (b and c). Specifically, the intercorrelations between idea realization and idea generation factor, as well as between the last one and idea promotion were above 1. This result indicated that a multi-dimensional structure was invalidate. In the same way, the average variance extracted (AVE) and the composite reliability (CR) indixes showed no evidence to support the discriminant-convergent validity and the reliability of the supposed three dimensions of innovative behavior. Given these results, it is concluded that the uni-factorial measurement model is the most suitable solution in the analyzed sample. This model showed adequate composite reliability, as well as configural and metric invariance between men and women. In line with own Janssen (2000) the dimensions may be to combine additively to create an overall scale of innovative work behavior. Future work should explore the nomological network of innovatiwe work behaviours in order to obtain more robust evidence for construct validity. Although some further research is needed, this study has tried to provide evidence of internal validity of a simple and pratical measure that is suitable for different work contexts, especially when innovative efforts are needed from all employees. It can be used in further research aimed to enhance our understanding of individual innovation.

Psychometric Validation of the Brief Alcohol Social Density Assessment in an Adult Community Sample.

Individuals' social networks exert a strong influence on alcohol use, but valid assessment of network drinking behavior is typically lengthy and high in participant burden. The aim of this study was to validate the Brief Alcohol Social Density Assessment (BASDA), an efficient measure of perceived alcohol use within a person's social network, in a sample of adult drinkers from the general community. Specifically, the convergent, criterion-related, incremental validity and internal validity were investigated by examining the BASDA in relation to other established measures of drinking motives, weekly drinking level, and severity of involvement. Participants were 903 (56% female) adults who reported drinking in the last year and who completed the BASDA, the Drinking Motives Questionnaire, the Daily Drinking Questionnaire, and the Alcohol Use Disorders Identification Test (AUDIT). Significant positive correlations were found between the BASDA and drinking motives, drinking quantity, and the AUDIT (rs=0.21 to 0.51, ps<0.001), providing support for convergent validity. There was a significantly higher BASDA score for those scoring at or above an AUDIT cutoff for hazardous drinking (p<0.001), providing support for criterion-related validity. Finally, beyond motives and covariates, the BASDA was significantly associated with total AUDIT score (ΔR2 =0.09, p<0.001), indicating its additive contribution and providing support for incremental validity. Confirmatory factor analysis revealed excellent fit, and all items significantly loaded onto a single factor (p<0.0001), providing evidence of internal validity. The resulting alcohol social density latent variable was significantly and robustly associated with drinks per week and AUDIT total score. These findings provide further support for the BASDA as a valid and efficient measure of social network alcohol density for understanding social influences on alcohol misuse.

Development and validation of a 21st Century Skills Assessment: Using an iterative multimethod approach

AbstractIn response to the call for more rigorously validated educational assessments, this study used an iterative multimethod validation process to develop and validate outcomes from the 21st Century Skills Assessment global rating scale. Qualitative and quantitative data sources were used to inform four types of validity evidence: content, response process, internal structure, and consequential. STEM experts and typical assessment users provided support for content, response process, and consequential validity evidence through focus group interviews and cognitive interviews. Psychometric findings suggest strong internal validity evidence for the assessment through high measures of internal consistency reliability, intra‐rater agreement, and test–retest reliability measures.

Avaliação de Estudantes quanto a Práticas Docentes para Criatividade na Pós-graduação

The purpose of this study was to investigate how master and doctoral students’ assess the extension to which graduate education professors favor the development of creativity in the classroom. The participants were 371 students from public and private universities. It was used the Inventory of Teaching Practices for Creativity in Higher Education, originally prepared for undergraduate students and adapted to the graduate context. The study also aimed to obtain evidence of internal validity of this instrument. The Inventory was valid to measure a dimensionality of three factors: General, Encouragement of New Ideas, and Interest in Student Learning. The third factor did not present an appropriate level of reliability. The results indicated a positive evaluation of graduate students with respect to their professors, as well as significant differences regarding the students’ evaluation considering type of institution and whether student works or not.Keywords: creativity, teaching, graduate level, scale.

Zarit Caregiver Burden Interview: Evidências de Validade para a População Brasileira de Cuidadores de Idosos

A Zarit Caregiver Burden Interview (ZBI) avalia a sobrecarga de cuidadores de idosos. Porém, ainda carece de evidências de validade para o contexto brasileiro, com cuidadores de idades variadas. Neste estudo, foram verificadas evidências de validade do ZBI –interna e baseadas nas relações com qualidade de vida e depressão–. No Estudo 1, participaram 285 cuidadores de idosos com idades entre 18 e 87 anos, que responderam à ZBI. No Estudo 2, 205 dos cuidadores do Estudo 1 também responderam ao Inventário de Depressão de Beck e à Escala de Qualidade de Vida. No Estudo 1, foram testados três modelos por meio de uma análise fatorial confirmatória e o alfa de Cronbach. No Estudo 2, foram medidascorrelações de escores na ZBI com escores de depressão e qualidade de vida. O modelo de três fatores se mostrou o maisadequado, apresentando bons índices de consistência interna. Foram observadas correlações negativas entre a ZBI e qualidade de vida e positivas com depressão. Portanto, este estudo contribuiu para o processo de validação da ZBI no Brasil, com cuidadores de faixa etária ampla.

Two faces of employee inactivity: Procrastination and recovery

The present study aims to investigate whether the two kinds of employee inactivity, namely procrastination and recovery at work can be empirically distinguished. Online survey data collected from 116 Dutch white-collar employees were analyzed in terms of (a) the factorial distinction of recovery and procrastination (internal validity evidence) and (b) their association with performance and work engagement (external validity evidence). The results of confirmatory factory analysis revealed the best fit for the model in which recovery and subdimensions of procrastination (soldiering and cyberslacking) loading to separate latent constructs. Moreover, recovery was positively related to work engagement and performance, whereas, procrastination did not show any significant relationship to these two concepts. In sum, findings of this study suggest that employee non-work related activity during work hours can be differentiated empirically. These results enhance the workplace procrastination literature by providing further validation evidence.